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during the suaaer of 1973 in 50 school districts in Hev York State. 
Project Alert had a tvo*fold purpose: to increase the 
diagnostic/prescriptive skills of selected classrooa teachers and to 
increase the leadership capabilities of the selected teachers so that 
they night function as inservice facilitators during the following 
school year. Evaluation of the inservice progran was accoaplished 
through the use of teachers* self-ratings* directors* ratings* and 
■onitors^ ratings. Analysis of the pre- and post-self-ratings 
suggests that teacher coapetency increased between the start of the 
inservice progras and its teraination. (TO) 



ERIC 



lOUCATtON A Wit^Att 
NATlOMAW iNlTlTUTt 0^ 
lOUCATlOM 

TMiS OOCUMCNt MA$ SEEN REPftO 
OUCCO EXACTLY AS RECEIVED PftOW 
THE REASON OR OR0ANl2ATlON CftlGiN 
ATlNOiY JOINTS or viEM Oft OPINIONS 
STATED 00 NOT NECESSARILY RE^RC 

SENTOPPlClALNATlONAHNSTlTUTEOP ^ ^ 

EDUCATION POSITION OR POLICY THOM/IS P» FITZGERALD 

New York State Education Department 

RICHARD M, CLARK 
State University of New Ycrk at Albany 

BEST COPY AVAILABLE 



PERMISSION TO REPWODUCC TMiS COPV 
RlGMteo MATERIAL HAS BFCN CiRANTSD BV 

Thomas P« Fitzgerald 
Richard M. Clark _ 

TO rWC AND ORGAM/ATlOfy^S OPf RATING 
UNDER AGRFtMENTS WViTm THfc NATIONAi IN 
STITUTE OF EOUCAT!'JN FURTnen R£PRO> 
OUCTlON OUTSlDt THt CRiC SVSTCM Rfc. 
QUiRtS PERMISSION OF THt COPYRIGHT 
OMNFR 



PROCESS EVALUATION FOR INSERVICE TRAINING^^ 
Many school districts have increased their inservice activities 
for teachers and have explored a variety of methods in presenting pro« 
grams. Both product and process evaluation are necessary to provide 
che necessary information on which to judge the effectiveness of such 
:raining programs* The ultimate goal of most Inservice programs may 
be viewed as a change in the product! in this case» the improved read- 
ing achievemenc of children* The more immediate objective may be 
viewed as a process change in which the teacher demonstrates a cog* 
nitlvet affective or behavioral development as a result of the inser* 
vice training* 

Little effort has been made to evaluate many inservice programs* 
When an evaluation has been made, it has often taken the form of a 
Sf ries of questions asking the participants to indicate the perceived 

^Paper presented at the National Reading Conference* Kansas City, 
Dec* 1974* 



degree of enjoyment, interest, or value in the inservice activity* 
Few programs have used a product evaluation in an effort to link stu- 
dent achievement to teacher training. When inservice programs have 
been subjected to cither a product or process evaluation in reading, 
the results have tended to receive criticism based on their reported 
procedures and instrumentation (Mobury, 1972; Herrick, 1957). Gen- 
erally, training programs have not included Judgements concerning 
changes in the classroom performance of teachers. More frequently, 
the evaluation of reading inservice has focused on attitudes (Schirmer 
and Navarre, 1968; Brown and Dutton, 1972), cognitive growth (Wall, 
1965) and simulation techniques (Kasden and Kelly, 1969). 

Another approach to assessing process change is the rating of 
classroom performance estimates made by supervisors (Sawyer and Taylor, 
1968) and by teacher self- rating (Heilman, 1966; DeCarlo and Cleland, 
1968). Such self-rating by teachers was designed to reflect changes 
in professional knowledge about the reading process or changes in at- 
titudes toward reading techniques. However, previous studies have 
not employed teacher self-rating to assess classroom applications of 
the topics discussed during the inservice. 

Self-ratings of teacher classroom behavior have not been used 
in conjunction with external ratings of the teacher's classroom 
behavior. 

Purpose 

The purpose of this study was to compare three methods of assess- 
ing teacher classroom behavior in the teaching of reading. A rating 
scale was constructed based on behavioral definitions of stsps in a 
diagnostic-pr«3scriptive approach to reading instruction. The behavior 



3 



assessed was then related to the objectives of an in-service program. 
Pre and post workshop ratings were made by teachers on a self •assess* 
ment basis and by workshop directors who were working closely with 
the teachers involved. Post workshop ratings were made by external 
observers on the basis of classroom visits and teacher interviews. 
The questions considered In this study are: 

(1) To what degree do ratings of teaching skills in reading 
change from pre-workshop to post-workshop? 

(2) What are the relationships of the three methods of assess- 
ing teacher skill? 

(3) What are the inter-relationships of the four skills in 
teaching reading which are being assessed for each pro- 
cedures used? 

(4) What relationships exist between a teacher's academic 
preparation in reading and the various assessments of 
reading competency? 

Procedure 

An Inservlce program, Project Alert, was conducted during the 
Summer of 1973 in 50 school districts in New York State. The program 
was initiated and coordinated by the Reading Bureau of the State Edu- 
cation Department. Each of the fifty district programs involved was 
administered by a project director who had received assistance from 
the Reading Bureau's staff and had received a six package kit of in- 
service material which was used as the basic inservlce tool. Project 
Alert had a twofold purpose: 

(1) to increase the diagnostic/prescriptive skills of selected 
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classroom teacher st and 
(2) to increase the leadership capabilities of the selected 
teachers so that they might function as inservice facili- 
tators during the 1973-74 academic year. 
Data for this study were compiled from 383 teachers participating 
in the inservice program. Each teacher recorded a rating of their 
classroom skills during the first (pre*inservice) and last (post-in- 
service) weeks of the program. Each project director used the same 
rating scale to estimate the classroom performance of individual tea- 
chers during the second and fourth weeks of training. An outside 
monitor observed randomly selected teachers and estimated their per- 
formance using the same rating scale during the fourth week. 

Assessment Instrument 

The assessment instrument (Appendix A) was constructed on a five 
point scale ranging from little behavioral evidence (1 on the scale) 
to considerable behavioral evidence (5) of skill demonstrated irithln 
a classroom. The set of five definitions was organized to reflect 
reasonably discrete steps leading from an absence of a skill to a 
highly developed skill display within the classroom. 

Such a set of five definitions was constructed in each of four 
areas: diagnosis, prescription* classroom management (execution) 
and evaluation. 

A panel of reading administrators suggested stages In the develop- 
ment of each of the four areas. The project evaluators then formu* 
lated behavioral statements to reflect these stages. The scales were 
modified in language for the self-rating and the observation. The 
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scales also were organized so that teachers an! administrators could 
have a common understanding of the described operations. 

Resul ts 

To determine the consistency of the ratings across rating groups 
of teachers, directors and monitors and the consistency of ratings 
across the time span of the Inservlce program, the mean ratings of 
each group were computed. These dat.- are displayed In Table 1. The 
teacher self -ratings before the Inservlce were consistently higher 
than the estimate made by the directors during the second week. The 
lowest ratings from teachers occurred In the category of diagnosing 
while the directors rated the evaluation skills of the teachers as 
the lowest demonstrated skill. Both groups rated the classroom man« 
agement (execution) skill as the most highly developed. 



(Insert Table 1 about here) 



The second series of ratings during the final weeks of the In- 
service program, revealed considerable variations among rating groups. 
The directors reported the highest ratings while the monitors recorded 
the lowest rating. The monitors' estimates were considerably lower 
than either the teacher's self-rating or the directors rating. 

The data presented In Table 1 Indicate that changes In the ratings 
occurred during the Inservlce program In teachers' self- ratings and, 
more dramatically, In the ratings made by the directors. For each 
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group the greatest rating change occurred in the diagnosis category 
with the smallest change occurring in the execution category. 

(Insert Table 2 about here) 

The inter-correlational matrix presented in Table 2 shows the 
inter* relationships between the various ratings o£ teacher behavior. 
In their pre-workshop assessments teachers clearly do not make inde- 
pendent ratings of the four skills which they are asked to rate. 
Correlations range from .49 to .82. (Correlation cluster 1-4). Post 
workshop ratings of teachers are also inter- related, but to a lesser 
def,ree, with correlations ranging from .43 to .57. (Clust'tr "^-S) . 
While positive correlations continue » there is a relatively small 
relationship bet«feen the ratings teachers give themselves pre-work- 
shop and post-workshop. Correlations range from .07 to .38. 
(Cluster 1-4 related to S-8). 

Directors show a consistent pattern of interrelationship in 
their ratings of the four skills both within the two periods when 
they made ratings and between the periods. (Cluster 9-16). However, 
the relationship between the ratings of the Director and the ratings 
of the teachers are generally positive but small. For exan^le, Lhe 
inter-correlations of items 5-8 with 13«-16 represents the relation- 
ships between the teachers' post workshop rating and the Directors' 
post-workshop rating. Correlations range from .09 to .28. 

Inter-Gorrelatf.ons of monitors' ratings in the four areas sugi^es;. 
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ih&t monitors made almosi: completely a global rating. Correlations 
range from .91 to .94 (Cluster 17«20). Further, extremely low rela- 
tionships are seen between teacher self-ratings and monitor ratings. 
(Cluster 1-8 and 17-20). However, modest but significant positive 
ratings were obtained between Directors' and monitors ratings. (Clus- 
ter 9-16 and 17-20). 

The fourth question in this study was the relationship between 
the teachers' self- rating and their academic background in reading. 
It might be expected that teachers with more advanced preparation in 
reading would evaluate their skills higher than those with little for- 
mal course work in reading. 

(Insert Table 3 about here) 

Table 3 summarizes the m tings of two groups of teachers: one 
wi;:h advanced degrees in reading and one with a single undergraduate 
course in reading or no formal preparation. As expected, the group 
with advanced training in reading consistently indicated higher self- 
ratings when compared to the group with little course work. Teachers 
with little training recorded positive changes in their ratings from 
the pre- to post assessment. Unexpectedly, the teachers with ad- 
vanced degrees tended to lower their post-assessment of the class* 
room behavior possible indicating that their graduate program did not 
concentrate on classroom applications of the four skill areas that 
were rated. 
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Conclusions 

The use of teacher self^ratings coupled with ratings from other 
sources appears to be a reasonable approach to determining behavioral 
changes generated through an inservice program* The rating from 
teachers, inservice directors and outside monitors produced different 
estimates of classroom behaviors, and these ratings did differ be- 
tween the start of the Inservice program and its termination* 

The two pre^post ratings suggest that teacher competency in^ 
creased and a small positive relationship was found between the two 
means of assessment* In this study it was assumed that the four spe* 
cific skills to be assessed might be somewhat related, but chat they 
were also to some degree independent* For example, teachers very 
good in diagnosis might generally be above average in evaluation, 
but the two skills would not be highly correlated* Self assessments 
of teachers* and monitors* ratings inter^relate to a moderate degree, 
suggesting that some independence might exist for the four skills 
rates* Monitors*, however, failed to make any meaningful distinction 
among the four areas* Further, monitors* ratings had almost a 
zero correlation with teacher self*ratings, and relatively small re» 
lationships with directors* ratings* 

Supposedly, the three measures used in this study, self-rating, 
directors* rating and monitors ratings were all measures of the same 
behaviors* Empirically, the three measures are not the same* Further 
work needs to be done in order to determine which assessment technique 
may be the most valid for a specific purpose* 

The teacher* 8 rating pattern appeared to be related to academic 
background especially for the pre<^inservice assessment* Surprisingly, 



during the post assessment, a smaller number of teachers with the 
higher academic background in reading reported estimates o£ their 
skills lo be at the highest ranking (5). while the number of teachers 
with low academic background in reading generally increased in re- 
porting higher self estimates. Perhaps an effect of the inservlce 
program was to reveal to teachers more of what could be known about 
the diagnostic-prescriptive procedure. If this explanation is cor- 
recu, then teachers with considerable academic background In an 
area, would enter a training program thinking that they had little 
need for further work with techniques for improving reading in- 
struction. During the inservlce program they might have discovered 
a need for additional ways to apply their knowledge within their 
classroom. This explanation is being tested in further research. 
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Appendix A 



Descriptive Categories for Ratlnif 

Dlettnosis 

1. No real diagnostic techniques in evidence except in such general 
ternt as "children lack reading akllU." 

2. Diagnostic techniques very general . Teacher has some data In- 
dicating general reading weaknesses of soiae nenbers of the group. 
However* it is quire possible that some individuals don't need 
the program which is specified. 

3. Diagnostic techniques seem to be rather general. DatA are 
available to indicate the general needs of a group, but it is 
not clear that every individual in the group has the need 
specified. 

4. Evidence of some individualized diagnostic procedures. Teacher 
is able to show specific procedures, but perhaps for only part 
of the group. For some children, a rather general diagnostic 
approach has been used. 

5. Clear evidence of an individualized diagnostic procedure. 
Teacher is able to show specific procedures which have been 
used which seem to pinpoint the needs of each individual in 
the group. 

Prescription 

1. Materials and activities used seem not to be especially relevant 
to the needs of many individuals in the group. 

2. No clear rationale seems to exist which relates the materials 
and activities of the program to the needs of the learners. The 
material and activities may well be appropriate, but such ap- 
prt^prlateness is assumed rather than reasoned. 

3. Materials and activities used seem to be based on a group 
assessment of need, not an individual diagnostic approach. 
Materials and activities appear to be well suited to this group 
diagnosis. 

4. The teacher seems to have a clear rationale for the selection 
of materials and activities for some children, but some members 
of the group appear to be working on material trttich is not based 
on a specific prescription of need. 

5. Teacher seems to have a clear ratlonaU which she has used to 
match each child to the materials and activities with which he 
is engaged. Thus, the teacher explains the work group. Materi- 
als selected to be appropriate for the purposes which the teecher 
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Execution 

1. Many problems in carrying out program as planned. Activities 

are often not carried out by children^ because of material bottle- 
necks, lack of effective organization^ or similar reasons* 

2. Some problems encountered in carrying out program plans. Some 
children seem not to know what they are expected to do. At 
times, they disrupt other children. Frequent delays Interrupt 
the flow of the day^s activities. 

3. A rather large amount of time seems to be taken by the mechanics 
of the operation. Children often have to wait for material to 
be ready. There may be use of "busy work" material primarily 

Lo keep children occupied. However, teacher-pupU rapport gen- 
erally satisfactory, no general disruption of program* 

4. A generally good learning environment. Children are usually 
occupied, and appropriate material Is readily available. At 
Limes, children are forced to wait for teacher to be available 
so chat Lht^y may move to new activity, but usually children move 
smoothly from one activity to another. 

5. Teacher exhibits a great deal of skill In having appropriate ma- 
terial is readily available for children, in arranging a good 
learning environment, in moving children from one activity to 
another, in maintaining rapport with children. 

Evaluation Process 

1. A fixed program is offered the children, perhaps on the basis 
of diagnostic techniques established before the program began. 
However, no effort is apparent wl thin the program , to determine 
whecher the children, even as a group, are meeting the objectives 
set for them. Even though pre- post- testing might be used, the 
data are so general that little or no evaluation of specific pro- 
gram materials is possible. 

2. Evaluation of the program seems to be largely on a pre- post 
basis and largely on a group basis. Thus, although children may 
be programmed on the basis of assessed needs, little or no 
evaluation activity goes on wi thin the group . After the program 
is ended, one might know how children have benefited, but such 
information would not be used during the program. 

3* Some evaluation activities occur wi thin the instructional period. 
Some evidence that at least groups of children are making progress, 
within the program, and that such evidence is being used to make 
program decisions. 

U. Clear evidence that evaluation of individual children is taking 



place on a regular basis and that these evaluations are used to 
determine the next set of activities for some individuals* At 
times evaluations might be somewhat delayed or not up to date 
for individuals* 

Clear evidence that frequent checks are made of the progress of 
almost every child* Evidence from such evaluations is promptly 
used to determine the next set of activities in which the indl« 
vldual should be engaged* 
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Table 1 

Mean Racings of Teachers' Diagnostic 
Prescriptive Abilities 



ABILITIES 



Ratings By Diagnosis Prescription Execution Evaluation 



Teacher - Pre sujamer 3.48 

Director • 2 weeks 3.37 

Teacher • Post summer 3.79 

Director - 4 weeks 4.24 

Monitor - 4 weeks 2.22 



3.52 3.95 3.69 

3.43 3.71 3.23 

3.80 3.98 3.88 

4.11 4.25 3.94 

2.10 2.14 1.89 
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Table 2 

Correlations Between Various Assessments 
of Diagnostic^Prescriptive Ability 



Assessments By 
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Teachers' Pre 
I. Diagnosis 


1.00 


.82 


.49 


.64 


.38 


.35 


.08 


.26 
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.20 
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3. Execution 






1 .00 
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.00 
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4. Evaluation 
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Teachers* Post 
5. Diagnosis 
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.35 


.52 


.31 
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6. Prescription 












1.00 
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.57 


.23 
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7. Execution 














1.00 
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.0. 


8. Evaluation 
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Directors' two week 
9. Diagnosis 


















1.00 


.7 



1 0. Prescripcion 

1 1 . Execucion 



1 2 > Evaluation 
Directors' four week 
13. Dia/gnosis 



1 4. Prescription 



15. Execution 



16 • Evaluation 



Monitors' rating 
17* Diagnosis 



1 8. Prescription 

19. Execution 



20. Evaluation 
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Table 2 continued 
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10. Prescription 
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11. Execution 
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TABLE 3 

Concrasc of Pre-* and Po8t<- Self*Rating8 of Teachers 
wich che Most and Least Academic Background 

In Reading 



Teachers with Advanced Degrees Teachers with the Least Academic 

in Reading Background in Reading 



Skill 




I 2 


Rating 
3 4 


5 
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Rating 
2 3 


4 
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Pre 


0 0 
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17 


33 


15 


46 


72 


83 
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25 
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11 


37 
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27 
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42 
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37 
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